Visualizing NYC Energy and Gas statistics
By: David Zhang and Hamim Choudhury
A general Overview of the original city Datasets
<class 'pandas.core.frame.DataFrame'> RangeIndex: 103259 entries, 0 to 103258 Columns: 265 entries, Calendar Year to Neighborhood Tabulation Area (NTA) (2020) dtypes: bool(1), float64(8), int64(8), object(248) memory usage: 208.1+ MB None
This amount of data is unnecessary for the scope of our project. Also many specific data points are stored as NULL or "not available". So we will parse through the data and extract the most interesting and useful columns.
KEEP_COLS = [
"Property ID",
"Property Name",
"Address 1",
"Postal Code",
"Year Built",
"Primary Property Type - Portfolio Manager-Calculated",
"Latitude",
"Longitude",
"Borough",
"Census Tract (2020)",
"Neighborhood Tabulation Area (NTA) (2020)",
"Property GFA - Calculated (Buildings) (ft²)",
"Site Energy Use (kBtu)",
"Weather Normalized Site Energy Use (kBtu)",
"Electricity Use - Grid Purchase (kWh)",
"Natural Gas Use (therms)",
"Site EUI (kBtu/ft²)",
"Weather Normalized Site EUI (kBtu/ft²)",
"ENERGY STAR Score",
"Total (Location-Based) GHG Emissions (Metric Tons CO2e)",
"Total (Location-Based) GHG Emissions Intensity (kgCO2e/ft²)",
"Occupancy",
"Fuel Oil #2 Use (kBtu)",
"District Steam Use (kBtu)",
"Office - Weekly Operating Hours",
"Office - Number of Workers on Main Shift",
"Multifamily Housing - Total Number of Residential Living Units"
]
With our new parquet file we can start looking for any interesting patterns and findings
Before we start visualizing data, we have to understand what some of the data means.
GFA or gross floor area refers to the size of a building in square feet.
Site energy use and weather normalized site energy use refer to the total amount of energy spent.
Site EUI (energy use intensity) is a measure of how much energy a property uses compared to its size.
GHG (green house gases) emissions is how much gas is released converted to be represented in the equivalent mass of CO2.
Energy Star certification is a label given to buildings that operate more efficiently than 75% of other buildings.
def clean_numeric_column(series):
"""Convert string columns with commas to numeric"""
if series.dtype == 'object':
return pd.to_numeric(series.str.replace(',', ''), errors='coerce')
return series
numeric_cols = [
'Property GFA - Calculated (Buildings) (ft²)',
'Site Energy Use (kBtu)',
'Weather Normalized Site Energy Use (kBtu)',
'Electricity Use - Grid Purchase (kWh)',
'Natural Gas Use (therms)',
'Site EUI (kBtu/ft²)',
'Weather Normalized Site EUI (kBtu/ft²)',
'ENERGY STAR Score',
'Total (Location-Based) GHG Emissions (Metric Tons CO2e)',
'Total (Location-Based) GHG Emissions Intensity (kgCO2e/ft²)',
'Fuel Oil #2 Use (kBtu)',
'District Steam Use (kBtu)'
]
for col in numeric_cols:
if col in df.columns:
df[col] = clean_numeric_column(df[col])
# Create shorter column names for easier plotting
df['GFA'] = df['Property GFA - Calculated (Buildings) (ft²)']
df['Site_Energy'] = df['Site Energy Use (kBtu)']
df['Weather_Norm_Energy'] = df['Weather Normalized Site Energy Use (kBtu)']
df['Electricity'] = df['Electricity Use - Grid Purchase (kWh)']
df['Gas'] = df['Natural Gas Use (therms)']
df['Site_EUI'] = df['Site EUI (kBtu/ft²)']
df['Weather_Norm_EUI'] = df['Weather Normalized Site EUI (kBtu/ft²)']
df['ENERGY_STAR'] = df['ENERGY STAR Score']
df['GHG_Total'] = df['Total (Location-Based) GHG Emissions (Metric Tons CO2e)']
df['GHG_Intensity'] = df['Total (Location-Based) GHG Emissions Intensity (kgCO2e/ft²)']
df['Property_Type'] = df['Primary Property Type - Portfolio Manager-Calculated']
# Remove extreme outliers for better visualization
df_clean = df[
(df['GFA'] > 0) &
(df['Site_EUI'] > 0) &
(df['Site_EUI'] < df['Site_EUI'].quantile(0.99))
].copy()
print(f"Original data: {len(df)} rows")
print(f"Cleaned data: {len(df_clean)} rows")
Now that we have a general understanding of what the data represents, we can now start analyzing any trends and statistics.
We can first start with a general visualization of the distribution of our data.
# 1. Distribution of Properties by Borough (Bar Chart)
borough_counts = df_clean['Borough'].value_counts().reset_index()
borough_counts.columns = ['Borough', 'Count']
# Create Plotly bar chart
fig_borough = px.bar(
borough_counts,
x='Borough',
y='Count',
title='Distribution of Properties by Borough',
labels={'Count': 'Number of Properties', 'Borough': 'Borough'},
color='Borough',
template='plotly_white',
autoresize=True
)
# Order the bars by count (largest first)
fig_borough.update_layout(xaxis={'categoryorder': 'total descending'})
fig_borough.show()
{% raw %} {% endraw %}
{% raw %} {% endraw %}
{% raw %}}script type="text/javascript"> window.PLOTLYENV=window.PLOTLYENV || {}; if (document.getElementById("fbbdb1c2-d7ce-464e-ab99-de6380a8325f")) { Plotly.newPlot( "fbbdb1c2-d7ce-464e-ab99-de6380a8325f", [{"hovertemplate":"Property Type=%{x}\u003cbr\u003eNumber of Properties=%{text}\u003cextra\u003e\u003c\u002fextra\u003e","legendgroup":"","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"","orientation":"v","showlegend":false,"text":{"dtype":"f8","bdata":"AAAAAEB97EAAAAAAAHK6QAAAAAAA47RAAAAAAAAwnUAAAAAAALiSQAAAAAAAaI9AAAAAAADwiUAAAAAAAPCIQAAAAAAAKIdAAAAAAADAhEAAAAAAANiBQAAAAAAAcIBAAAAAAACwfUAAAAAAAFB9QAAAAAAAYHxA"},"textposition":"outside","x":["Multifamily Housing","Office","K-12 School","Hotel","Non-Refrigerated Warehouse","Manufacturing\u002fIndustrial Plant","Retail Store","Self-Storage Facility","Mixed Use Property","College\u002fUniversity","Worship Facility","Residence Hall\u002fDormitory","Distribution Center","Senior Living Community","Parking"],"xaxis":"x","y":{"dtype":"i4","bdata":"6uMAAHIaAADjFAAATAcAAK4EAADtAwAAPgMAAB4DAADlAgAAmAIAADsCAAAOAgAA2wEAANUBAADGAQAA"},"yaxis":"y","type":"bar","texttemplate":"%{text:,}"}], {"template":{"data":{"histogram2dcontour":[{"type":"histogram2dcontour","colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]}],"choropleth":[{"type":"choropleth","colorbar":{"outlinewidth":0,"ticks":""}}],"histogram2d":[{"type":"histogram2d","colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]}],"heatmap":[{"type":"heatmap","colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]}],"contourcarpet":[{"type":"contourcarpet","colorbar":{"outlinewidth":0,"ticks":""}}],"contour":[{"type":"contour","colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]}],"surface":[{"type":"surface","colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]}],"mesh3d":[{"type":"mesh3d","colorbar":{"outlinewidth":0,"ticks":""}}],"scatter":[{"fillpattern":{"fillmode":"overlay","size":10,"solidity":0.2},"type":"scatter"}],"parcoords":[{"type":"parcoords","line":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scatterpolargl":[{"type":"scatterpolargl","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"#E5ECF6","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"scattergeo":[{"type":"scattergeo","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scatterpolar":[{"type":"scatterpolar","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"scattergl":[{"type":"scattergl","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scatter3d":[{"type":"scatter3d","line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scattermap":[{"type":"scattermap","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scattermapbox":[{"type":"scattermapbox","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scatterternary":[{"type":"scatterternary","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"scattercarpet":[{"type":"scattercarpet","marker":{"colorbar":{"outlinewidth":0,"ticks":""}}}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"white","linecolor":"white","minorgridcolor":"white","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"white","linecolor":"white","minorgridcolor":"white","startlinecolor":"#2a3f5f"},"type":"carpet"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}],"barpolar":[{"marker":{"line":{"color":"#E5ECF6","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"pie":[{"automargin":true,"type":"pie"}]},"layout":{"autotypenumbers":"strict","colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"hovermode":"closest","hoverlabel":{"align":"left"},"paper_bgcolor":"white","plot_bgcolor":"#E5ECF6","polar":{"bgcolor":"#E5ECF6","angularaxis":{"gridcolor":"white","linecolor":"white","ticks":""},"radialaxis":{"gridcolor":"white","linecolor":"white","ticks":""}},"ternary":{"bgcolor":"#E5ECF6","aaxis":{"gridcolor":"white","linecolor":"white","ticks":""},"baxis":{"gridcolor":"white","linecolor":"white","ticks":""},"caxis":{"gridcolor":"white","linecolor":"white","ticks":""}},"coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]]},"xaxis":{"gridcolor":"white","linecolor":"white","ticks":"","title":{"standoff":15},"zerolinecolor":"white","automargin":true,"zerolinewidth":2},"yaxis":{"gridcolor":"white","linecolor":"white","ticks":"","title":{"standoff":15},"zerolinecolor":"white","automargin":true,"zerolinewidth":2},"scene":{"xaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white","gridwidth":2},"yaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white","gridwidth":2},"zaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white","gridwidth":2}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"geo":{"bgcolor":"white","landcolor":"#E5ECF6","subunitcolor":"white","showland":true,"showlakes":true,"lakecolor":"white"},"title":{"x":0.05},"mapbox":{"style":"light"}}},"xaxis":{"anchor":"y","domain":[0.0,1.0],"title":{"text":"Property Type"},"tickangle":-45},"yaxis":{"anchor":"x","domain":[0.0,1.0],"title":{"text":"Number of Properties"}},"legend":{"tracegroupgap":0},"title":{"text":"Distribution of 15 Most Common Property Types"},"barmode":"relative","height":600,"showlegend":false}, {"responsive": true} ).then(function(){
var gd = document.getElementById('fbbdb1c2-d7ce-464e-ab99-de6380a8325f');
var x = new MutationObserver(function (mutations, observer) {{
var display = window.getComputedStyle(gd).display;
if (!display || display === 'none') {{
console.log([gd, 'removed!']);
Plotly.purge(gd);
observer.disconnect();
}}
}});
// Listen for the removal of the full notebook cells
var notebookContainer = gd.closest('#notebook-container');
if (notebookContainer) {{
x.observe(notebookContainer, {childList: true});
}}
// Listen for the clearing of the current output cell
var outputEl = gd.closest('.output');
if (outputEl) {{
x.observe(outputEl, {childList: true});
}}
}) }; {% endraw %}
{% raw %} {% endraw %}
{% raw %} {% endraw %}
{% raw %} {% endraw %}
Now we can answer some questions like ...
How does the age of a building correlate to its energy performance?
{% raw %} {% endraw %}
{% raw %} {% endraw %}
Now we can move onto analyzing how different attributes of buildings affect its emmisions and energy consumption.
Parameters like the building type, size, and occupancy can heavily affect the output.
{% raw %}{% endraw %}
{% raw %} {% endraw %}
{% raw %} {% endraw %}
{% raw %} {% endraw %}
{% raw %} {% endraw %}
{% raw %}